Moving object detection aims to separate the background and foreground of the video, however, the commonly used low-rank factorization methods are often difficult to comprehensively deal with the problems of dynamic background and intermittent motion. Considering that the skewed noise distribution after background subtraction has potential background correction effect, a moving object detection model based on the reliability low-rank factorization and generalized diversity difference was proposed. There were three steps in the model. Firstly, the peak position and the nature of skewed distribution of the pixel distribution in the time dimension were used to select a sub-sequence without outlier pixels, and the median of this sub-sequence was calculated to form the static background. Secondly, the noise after static background subtraction was modeled by asymmetric Laplace distribution, and the modeling results based on spatial smoothing were used as reliability weights to participate in low-rank factorization to model comprehensive background (including dynamic background). Finally, the temporal and spatial continuous constraints were adopted in proper order to extract the foreground. Among them, for the temporal continuity, the generalized diversity difference constraint was proposed, and the expansion of the foreground edge was suppressed by the difference information of adjacent video frames. Experimental results show that, compared with six models such as PCP(Principal Component Pursuit), DECOLOR(DEtecting Contiguous Outliers in the Low-Rank Representation), LSD(Low-rank and structured Sparse Decomposition), TVRPCA(Total Variation regularized Robust Principal Component Analysis), E-LSD(Extended LSD) and GSTO(Generalized Shrinkage Thresholding Operator), the proposed model has the highest F-measure. It can be seen that this model can effectively improve the detection accuracy of foreground in complex scenes such as dynamic background and intermittent motion.